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PREFACE 


This  document  was  prepared  by  the  Institute  for  Defense  Analyses  as  an 
interdivisional  effort  by  the  Operational  Evaluation  Division  and  the  Computer  and 
Software  Engineering  Division  for  the  Director  of  the  Joint  C4ISR  Battle  Center  and  for 
the  Joint  Chiefs  of  Staffs  Force  Structure,  Resources  and  Assessment  Directorate  in 
partial  fulfillment  of  the  task  “Assessments  of  the  Joint  C4ISR  Battle  Center.”  The 
object  was  to  present  a  conceptual  methodology  that  could  be  used  to  develop  the  scope 
and  objectives  and  to  assess  the  outcome  of  advanced  joint  warfighting  experiments. 

The  IDA  Technical  Review  Committee  was  chaired  by  Mr.  Thomas  P.  Christie, 
and  consisted  of  Dr.  Gary  Comfort,  Mr.  Dean  DeWolfe,  Dr.  William  Hurley,  LTG  Peter 
Kind,  USA  (Ret.),  and  Mr.  Richard  Miller. 
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SUMMARY 


This  document  suggests  an  approach  for  defining  independent  assessments  of  the 
results  of  joint  advanced  warfighting  experiments  undertaken  in  support  of  Joint  Vision 
2010.  The  joint  experiments  that  we  have  in  mind  are  those  pursued  in  development  of 
innovative  operational  doctrine  or  operational  concepts  to  be  employed  by  a  joint  task 
force,  a  unified  command,  or  similar  organization.  It  is  during  an  assessment  that  one 
may  see  the  possible  effects  on  operational  capability  and  military  utility;  independent 
assessments  would  have  the  goal  of  interpreting  the  operational  potential  of  new 
warfighting  concepts  free  of  the  biases  of  those  who  propose  the  capability  or  conduct  the 
experiments. 

Since  their  use  throughout  the  military  establishment  is  not  standard,  this 
document  first  presents  some  reasonable  definitions  of  the  meanings  of  innovation, 
experimentation,  and  jointness  as  used  in  the  context  of  joint  operations  and  the 
Chairman  of  the  Joint  Chiefs  of  Staff  s  vision  for  future  warfighting  capabilities.  Then, 
the  document  describes  the  nature  of  experiments,  their  purposes,  and  how  they  support 
the  development  of  advanced  warfighting  concepts  that  will  be  the  focus  of 
experimentation.  Following  that,  we  develop  a  structure  for  an  independent  assessment 
or  evaluation  of  an  advanced  warfighting  concept,  and  a  methodology  for  independently 
assessing  the  success  or  failure  of  experiments  dealing  with  future  warfighting  concepts 
are  developed. 

A.  PURPOSE  OF  AN  INDEPENDENT  ASSESSMENT 

One  of  the  characteristics  of  innovation  and  experimentation  is  the  value  in 
pushing  an  idea  to  the  breaking  point  and  learning  its  limits;  failure  is  a  virtue  in  this 
context.  Those  who  must  decide  what  to  acquire  or  what  new  warfighting  concept  to  try 
need  to  know  under  what  conditions  a  proposed  solution  can  be  expected  to  succeed  and 
under  what  conditions  it  may  fail.  Two  or  more  complementary  solutions  might  be 
required  rather  than  a  single  solution.  Assessing  the  value  of  warfighting  experiments 
thus  differs  substantially  from  evaluations  of  new  military  systems. 
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Those  who  have  been  assigned  responsibility  for  developing  a  specific  material 
solution  to  Service  needs  do  not  see  failure  as  an  attractive  option  and  usually  try  to  avoid 
or  reduce  the  risk  of  failure.  Clearly,  warfighting  experimenters  and  system  proponents 
would  approach  assessments  quite  differently.  Those  who  are  to  evaluate  the  outcomes 
of  warfighting  experiments  should  bear  in  mind  the  desirability  of  understanding  the 
conditions  of  failure  as  well  as  the  potential  conflicts  between  the  interested  parties. 

B.  DEFINING  THE  ASSESSMENT’S  SCOPE  AND  OBJECTIVES 

Systematic  planning  for  advanced  warfighting  concept  assessment  can  identify 
problems  and  organizational  requirements  and  help  the  force  planners  and  experimenters 
as  well  as  the  evaluators  reach  agreement  on  many  points.  Assessment  plans  are  not 
meant  to  be  directive  in  all  respects.  In  their  early  form,  they  are  conceptual  structures 
meant  to  motivate  and  focus  discussion  between  the  proponents  and  evaluators  on  such 
topics  as  evaluation  goals,  experiment  organization  and  design,  and  critical  resources 
needed  for  the  experiment  to  be  conducted.  Assessment  plans  also  facilitate  transition 
from  conceptual  thinking  to  defining  the  tasks  needed  to  begin  the  actual  experiment’s 
organization.  For  this  reason,  evaluators  need  to  be  involved  in  the  earliest  stages  of 
planning  a  joint  warfighting  experiment.  Figure  1  illustrates  the  steps  involved  in 
designing  an  advanced  warfighting  experiment  assessment  plan,  starting  with  the  basic 
goals,  then  identifying  key  issues  at  the  force,  unit,  and  system  levels,  and  finally 
identifying  the  main  experimentation  concepts  and  critical  resources  needed  to  implement 
the  proposed  experiment. 

An  assessment  plan  should  clearly  identify  those  issues  central  to  determining 
whether  the  new  warfighting  concept  has  potential  operational  advantages  and  will  have 
operational  utility  or  suitability  when  employed  by  a  force  in  the  field.  These  are  the 
Critical  Warfighting  Innovations  (CWls)  of  the  new  concept,  and  should  be  framed  as 
questions  phrased  to  pin  down  just  what  the  new  warfighting  concept  is  supposed  to 
provide.1  There  can  be  as  few  as  one  or  two  such  questions,  but  they  are  vital  because 
they  establish  the  tone  and  purpose  of  the  experiment  (what  the  new  capability  should 
demonstrate),  and  make  certain  that  everyone’s  expectations  are  similar.  They  are  the 
foundation  for  the  assessment  of  the  experiment’s  outcome. 


1  Some  innovations  will  be  designed  for  operations  other  than  war,  and  are  not  rightly  “warfighting.” 
Despite  this,  the  purpose  of  a  military  force  is  to  fight  in  the  nation’s  defense.  For  this  reason  we  have 
chosen  to  employ  the  term  "Critical  Warfighting  Innovations  throughout  the  paper  for  consistency. 
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Figure  1.  Developing  an  Experimentation  Assessment  Concept 


Because  innovation  is  a  continuing  and  cyclic  process,  an  assessment  plan  will  be 
a  living  document  that  will  change  over  time  as  more  information  about  the  experiment 
becomes  available.  The  plan’s  early  purpose  is  to  evoke  and  to  encourage  debate  about 
the  final  form  of  the  assessment,  to  avoid  as  many  last  minute  complications  as  possible, 
and  to  ensure  that  the  assessment  design  is  as  complete  as  possible.  When  fully 
developed,  the  plan  will  guide  the  final  evaluation  of  the  experiment’s  outcome. 


C.  INTENDED  OUTCOME  FROM  THE  ASSESSMENT  PLAN 

The  warfighting  experiment  assessment  plan  serves  to  highlight  certain  variables 
for  the  decision  makers:  the  issues  of  critical  importance  to  a  meaningful  independent 
assessment;  the  alternative  approaches  for  obtaining  the  needed  data;  and  the  complexity 
and  associated  costs  (in  broad  terms)  of  the  alternative  approaches.  The  assessment  plan 
should  assist  decision  makers  in  reaching  early  agreement  upon  the  principal  issues  to  be 
addressed  and  the  assessment  approach  to  be  employed. 


D.  FRAMEWORK 

In  order  to  develop  a  cohesive  assessment  of  any  new  warfighting  concept,  it  may 
be  necessary  to  make  a  number  of  assumptions  as  to  the  eventual  working  of  the  concept 
in  its  military  application  (either  combat  or  operations  other  than  war).  These 
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assumptions  may  center  on  understanding  the  operational  advantages  and  disadvantages 
of  the  application  or  on  assessing  the  quality  of  data  that  will  be  available,  not  only  to  the 
evaluators,  but  also  to  those  who  will  conduct  the  experiment,  and  the  evaluators’  ability 
to  take  actions  based  on  those  data.  To  do  this,  the  proven  approaches  used  for 
operational  evaluation  can  help  to  frame  the  questions  to  be  answered  and  then  to  guide 
the  analysis  of  the  experiment’s  outcome,  or  apparent  outcome.  For  operational  test  and 
evaluation,  these  approaches  are  divided  into  areas  dealing  with  operational  effectiveness 
(in  essence,  answering  the  question:  will  the  concept  or  system,  under  realistic 
operational  conditions  and  used  by  typical  military  personnel,  contribute  to  a  unit  s 
overall  mission  effectiveness?),  and  operational  suitability  (can  the  concept  actually  be 
employed  and  supported  by  the  military  system,  and,  if  not,  how  must  the  system  or 
concept  be  altered  to  accommodate  the  concept’s  implementation?). 

To  assess  proposed  advanced  warfighting,  however,  a  broader  and  less 
constrained  approach  than  the  operational  effectiveness  and  operational  suitability  criteria 
used  in  planning  operational  test  and  evaluation  for  a  major  system  acquisition  is  needed. 
The  more  free-ranging  goals  of  joint  warfighting  experimentation  or  experiments  requires 
that  we  adopt  a  flexible,  but  parallel,  approach  to  understanding  the  value  of  new 
concepts.  To  examine  advanced  warfighting  concepts,  we  believe  that  evaluators  should 
frame  their  concerns  in  two  assessment  areas:  operational  potential  and  operational 
supportability.  We  shall  use  these  latter  terms  to  describe  a  generic  evaluation 
framework.2 

The  operational  potential  issue  area  predominantly  involves  the  warfighting 
unit’s  ability  to  use  the  advanced  concept  to  organize,  coordinate  resources  and 
capabilities,  plan  missions,  integrate  components’  plans,  achieve  coordination  with  the 
various  approval  authorities,  issue  plans,  carry  out  the  plans,  survive,  and  recover. 

The  operational  supportability  issue  area  relates  to  the  ability  of  a  concept  when 
employed  by  a  joint  force  commander  to  support  that  commander’s  needs  in  terms  of  the 
human  requirements  for  its  use  and  the  user’s  ability  to  maintain  an  acceptable  level  of 
readiness,  to  interoperate  with  cooperating  systems  and  commands,  and  to  operate  in  a 


2  Another  reason  for  using  different  assessment  terminology  is  to  avoid  creating  the  impression  that  the 
concept,  system,  or  system-of-systems  in  the  experiment  is  involved  in  operational  test  and  evaluation. 
OT&E  is  a  much  more  formal  process  intended  to  satisfy  a  number  of  legal  requirements  and  to 
support  decisions  about  acquisition  milestones.  Should  the  object  of  the  experiment  eventually  be  part 
of  OT&E,  its  operational  effectiveness  and  operational  suitability  would  be  assessed  in  light  of  the 
then-current  requirements. 
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congested  environment  or  in  an  environment  where  an  enemy  is  trying  actively  to  disrupt 
operations.  In  addition,  the  experiment  should  yield  certain  data  necessary  for 
determining  how  manpower  intensive  and  logistically  demanding  the  concept  will  be 
when  brought  to  operational  use.  Moreover,  substantial  warfighting  changes  will  demand 
accompanying  structural  alterations  having  a  bearing  on  force  structure,  support  agencies, 
budgets,  training,  and  the  like.  The  nature  and  impact  of  these  alterations  must  be 
understood  as  well. 

1.  Assessment’s  Overall  Design 

The  experiment’s  design  (and  its  assessment)  should  be  built  upon  the  Critical 
Warfighting  Innovations  described  in  Chapter  III,  and  should  be  aimed  at  a  logical 
progression  of  topics  and  decisions: 

.  Focusing  on  the  warfighting  concept’s  accomplishments  as  opposed  to 
performance  parameters. 

.  Describing  unresolved  mission  deficiencies  yet  to  be  evaluated  or  to  be  the 
subject  of  follow-on  experiments. 

.  Describing  how  the  concept  that  is  the  subject  of  the  experiment  addresses 
potential  mission  deficiencies. 

.  Answering  the  question,  “Did  the  experiment  succeed  in  providing  sufficient 
data  from  which  to  reach  conclusions  about  the  subject’s  performance?” 

.  Understanding  what  is  adequate  for  experiment  success. 

.  Understanding  whether  the  subject  of  the  experiment  performed  adequately 
(that  is,  one  must  understand  how  to  measure  success  in  a  given  case). 

It  is  important  to  bear  in  mind  here  the  difference  between  the  experiment’s 
failure  to  provide  the  information  it  was  supposed  to  collect,  and  the  warfighting 
concept’s  failure  to  meet  expectations  as  the  subject  of  the  experiment. 

An  important  major  area  to  be  addressed  along  with  operational  potential  and 
operational  supportability  is  an  experiment’s  adequacy,  and  whether  or  not  the 
experiment’s  venue  or  environment  will  provide  the  information  needed  not  only  for  the 
independent  assessment,  but  also  for  the  final  decision  maker.  The  Assessment  Plan  is 
the  appropriate  mechanism  to  establish  the  hierarchical  structure  for  the  evaluation,  as 
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depicted  in  Figures  1  and  2.3  In  Figure  2,  the  most  all-encompassing  issues  to  be 
weighed  are  associated  with  the  topmost  block  (1.0),  to  determine  whether  the  command 
can  derive  an  advantage  from  employing  the  new  warfighting  concept.  If  the  assessment 
cannot  be  made  at  this  first  level  (and  likely  it  cannot),  then  subsequent  analyses  must 
explore  issues  farther  down  (Blocks  2.0  and  3.0)  to  determine  the  causes  and  the 
significance  of  the  advantages,  shortfalls,  or  problems  noted.  It  is  especially  important 
when  a  warfighting  concept  or  system  is  pushed  to  failure  while  in  an  experiment  or  test 
that  there  be  some  type  of  assessment  structure  so  that  one  may  follow  the  issues  of 
greatest  importance  until  one  is  able  to  define  the  point  at  which  failure  occurred  and  the 
reasons  why. 


The  numbers  in  the  boxes  correspond  to  the  evaluation  issues  discussed  in  Chapter  IV.  Blocks  2.0  Operational  Potential,  and 
3.0,  Operational  Supportability.  are  the  assessment  areas.  The  lower  level  blocks  define  the  specific  assessment  issues 


Figure  2.  Notional  Advanced  Warfighting  Concept  Assessment  Issues 


3  Of  course,  a  given  experiment’s  assessment  structure  would  depend  upon  the  nature,  extent  and 
purpose  of  its  subject’s  intended  use.  The  warfighting  innovation  example  on  which  the  assessment 
description  is  built  is  a  hypothetical  operational  concept  and  its  supporting  systems.  The  operational 
warfighting  concept  is  a  software  intensive,  distributed,  collaborative  planning  system  supported  by 
worldwide  communications.  The  topics  to  be  explored  in  an  actual  experiment  might  not  resemble 
those  in  this  example. 
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2.  Specific  Assessment  Areas 

Using  the  statements  of  need  or  the  requirements  expressed  by  the  potential  users, 
such  as  a  unified  command  or  a  JTF,  the  next  task  is  to  identify  a  set  of  assessment  issues 
(AI).  The  AIs  will  highlight  a  small  number  of  issues,  typically  fewer  than  ten,  whose 
satisfactory  resolution  will  provide  the  information  needed  to  determine  the  advanced 
warfighting  concept’s  operational  potential  and  operational  supportability.  From  the 
answers  to  questions  posed  by  the  AIs,  one  may  then  develop  an  assessment  of  the  value 
of  the  potential  new  capability. 

Assessment  issues  are  described  in  Chapter  IV,  as  are  the  ways  to  measure 
operational  potential  and  operational  supportability;  whenever  possible,  these  measures 
should  attempt  to  give  quantitative  information. 
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I.  INTRODUCTION 


This  document  suggests  an  approach  for  defining  independent  assessments  of  the 
results  of  joint  advanced  warfighting  experiments  undertaken  in  support  of  Joint  Vision 
2010.  The  joint  experiments  that  we  have  in  mind  are  those  pursued  in  development  of 
innovative  operational  doctrine  or  operational  concepts  to  be  employed  by  a  joint  task 
force,  a  unified  command,  or  similar  organization.  It  is  during  an  assessment  that  one 
may  see  the  possible  effects  on  operational  capability  and  military  utility;  independent 
assessments  would  have  the  goal  of  interpreting  the  operational  potential  of  new 
warfighting  concepts  free  of  the  biases  of  those  who  propose  the  capability  or  conduct  the 
experiments. 

Since  their  use  throughout  the  military  establishment  is  not  standard,  this 
document  first  presents  some  reasonable  definitions  of  the  meanings  of  innovation, 
experimentation,  and  jointness  as  used  in  the  context  of  joint  operations  and  the 
Chairman  of  the  Joint  Chiefs  of  Staffs  vision  for  future  warfighting  capabilities.  Then, 
the  document  describes  the  nature  of  experiments,  their  purposes,  how  they  support 
development  of  advanced  warfighting  concepts  that  will  be  the  focus  of  experimentation. 
Following  that,  we  develop  a  structure  for  an  independent  assessment  or  evaluation  of  an 
advanced  warfighting  concept,  and  a  methodology  for  independently  assessing  the 
success  or  failure  of  experiments  dealing  with  future  warfighting  concepts. 

As  used  in  this  document,  the  term  concept  refers  to  the  idea  or  proposal  that  is 
the  subject  of  the  warfighting  experiment.  A  concept  needs  to  contain  several  elements, 
including  reasons  why  the  concept  is  necessary  (usually  a  short  discussion  of  some 
mission  deficiency  that  needs  to  be  addressed);  what  is  to  be  accomplished  (in  terms  of 
goals  or  objectives,  stated  quantitatively,  if  possible);  the  operational  environment  in 
which  the  concept  is  expected  to  apply  and,  finally,  a  statement  of  how  it  is  anticipated  to 
be  put  into  practice.  Without  this  how  element,  it  is  little  more  than  a  mission  statement 
or  a  task  statement.  In  the  same  vein,  a  warfighting  concept  needs  to  include  some  way 
of  judging  or  knowing  when  one  has  achieved  one’s  warfighting  goal,  or  won  (or 
understanding  when  one  is  losing),  if  the  concept  were  to  be  employed. 
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Thus,  a  concept  is  an  abstraction  that,  when  developed  so  that  it  can  be  put  to 
military  use,  would  become  a  capability.  Stated  another  way,  a  concept,  through  a 
process  of  growth  and  development  may  become  a  military  capability.  Joint  warfighting 
experiments  are  an  integral  part  of  the  growth  and  development  process  leading  to  that 
capability. 

If  they  are  to  be  useful,  warfighting  experiments  must  be  designed  and  evaluated 
with  the  idea  in  mind  that  they  may  fail  to  validate  the  warfighting  concept  that  is  the 
subject  of  the  experiment.  If  failure  is  not  an  option,  it  is  not  an  experiment. 
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II.  DEFINING  INNOVATION,  EXPERIMENTATION, 

AND  JOINTNESS 


Calls  for  military  innovation  and  experimentation  currently  come  from  many 
quarters:  Congress,  Service  schools,  defense  analysts,  professional  writing  by  military 
practitioners,  and  as  justification  for  new  projects  and  undertakings.  The  intent  of  those 
seeking  to  promote  innovation  varies,  but  is  generally  directed  at  questions  of  how  or 
whether  to  change  the  style  and  structure  of  warfare  to  meet  anticipated  threats  of  the  21st 
Century  or  to  reduce  the  cost  of  military  forces  and  operations.  Such  innovation, 
however,  implies  different  military  forces  or  forces  employed  differently  from  those  we 
have  seen  in  the  Department  of  Defense  over  the  past  several  decades.  Some  possible 
changes  are  potentially  far-reaching,  involving  newly  developed  joint  operational 
doctrine  and  joint  force  employment  done  as  has  rarely  been  seen  since  World  War  II 
(and  even  then  jointness  as  currently  defined  was  unusual). 

If  such  changes  are  to  be  implemented,  the  process  carries  with  it  a  need  for 
independent  evaluation  of  the  new  operational  capabilities  to  ensure  that  U.S.  forces  do 
not  find  out  at  the  time  of  first  combat  employment  that  the  new  fighting  concept  or 
structure  will  not  stand  the  stress  of  the  encounter. 

A.  INNOVATION 

Despite  all  the  rhetoric  surrounding  the  need  for  newly  designed  forces  and  the 
innovation  and  experimentation  needed  to  devise  those  forces,  people  who  speak  of 
innovation  and  experimentation  in  a  military  operations  context  do  not  always  use  or 
intend  similar  meanings  for  those  words.  Often,  users  have  intuitive  definitions  in  mind, 
but  fail  to  state  clearly  what  it  is  that  they  mean.  Thus,  a  speaker  may  understand  what  he 
or  she  intends,  but  the  audience  is  not  so  clear  on  the  points  being  made.  As  a  result,  the 
ideas  these  two  words  convey  and  how  innovation  and  experimentation  should  be 
designed  and  achieved  are  not  necessarily  the  same  to  all  those  who  discuss  them.  In  the 
same  vein,  because  a  doctrine,  system,  or  technology  is  new  does  not  mean  that  it  is 
inherently  better  than  prior  or  existing  systems,  practices,  or  assets.  This  common 
association  between  new  and  better  represents  an  example  of  the  pitfalls  of  intuitive 
definitions. 


For  example,  several  senior  officers  and  members  of  Congress  recently  have  used 
the  term  innovate  in  ways  that  imply  achieving  improvements  or  expansion  of  capability 
or  reduction  of  costs  of  maintaining  a  military  force:  Senator  Dan  Coats  calls  for  leap- 
ahead  capabilities,”  General  Dennis  Reimer  seeks  “innovative  means,”  and  Lt.  Gen.  Paul 
Van  Riper  wants  the  Services  to  find  “new  capabilities.’  These  men,  however,  do  not 
always  define  their  objectives  in  terms  that  are  standard  and  quickly  understood  by 
others.  The  ambiguity  of  these  phrases  can  contribute  to  miscommunications  between 
the  participants  in  the  debate.  Recent  scholarship  has  added  to  our  understanding  of  just 
what  forces  bring  about  change  in  military  organizations.  Change  originates  from  the 
effects  of  military  culture,  from  civilian  efforts  to  alter  the  armed  services  driven  by  a 
variety  of  motivations,  and  from  technological  advances  coupled  with  a  general  readiness 
and  ability  to  apply  new  technology  to  warfighting,  among  others.1 

Despite  the  different  views  on  just  what  leads  to  innovation  in  the  military, 
innovation  is  a  continuing  process.  Ideas  are  tried  out  in  special  or  periodic  events  as 
experiments  based  on  some  theory  or  expectation,  often  in  a  cyclic  pattern,  with  each 
cycle  leading  to  new  understandings  of  operational  capabilities  or  requirements.  Usually, 
these  cycles  generate  new  questions  needing  new  sets  of  experiments  to  answer  them. 
These  cycles,  however,  establish  a  series  of  baselines,  with  each  baseline  containing  the 
results  of  the  latest  cycle  of  innovation  and  experimentation.  Figure  II- 1  illustrates  the 
cyclic  model  of  innovation.  In  this  model,  innovation  never  ends  and  no  goal  will  be  met 
fully  and  completely;  here,  results  of  small  experiments  “e”  feed  more  complex 
experiments  “E  ”  leading  to  yet  more  questions  to  be  answered.  As  improvements  are 
made  to  a  desired  capability,  the  military  and  geo-political  environment  changes  along 
with  technological  options. 


■  Air  operations  in  World  War  II  became  possible  because  of  remarkable  growth  in  aircraft  design  and 
manufacturing  (especially  the  use  of  machine  tools  for  metal  working).  Amphibious  warfare  depended 
not  only  on  development  of  the  employment  doctrine,  but  also  on  the  parallel,  but  separate,  design  of 
boats  and  ships  to  make  it  feasible.  See  Barry  R.  Posen,  The  Sources  of  Military  Doctrine  Ithaca,  NY: 
Cornell  University  Press,  1984;  Stephen  P.  Rosen,  Winning  the  Next  War:  Innovation  and  the  Modern 
Military,  Ithaca,  NY:  Cornell  University  Press,  1991;  and  Allen  R.  Millett  and  Williamson  Murray, 
Military  Innovation  in  the  Interwar  Period,  New  York:  Cambridge  University  Press,  1996. 
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For  purposes  of  this  paper,  and  to  achieve  uniformity  of  terminology  in  the 
discussion  of  design,  execution,  and  assessment  of  advanced  warfighting  experiments,  we 
shall  use  the  following  military-specific  definitions: 

.  Innovate,  v:  to  develop  new  doctrine,  systems,  or  technologies  that  represent 
a  departure  from  prior  or  contemporary  military  doctrine,  systems,  or 
technologies. 

.  Innovative,  adj:  representative  of  a  departure  from  prior  or  contemporary 
practices 

.  Innovation,  n:  a  new  military  doctrine,  system,  or  technology  that  represents  a 
departure  from  prior  or  contemporary  military  doctrine,  systems,  or 
technologies;  or  the  process  by  which  the  military  develops,  explores,  and 
possibly  implements  new  military  doctrine,  systems,  or  technologies  that 
represent  a  departure  from  prior  or  contemporary  military  doctrine,  systems, 
or  technologies. 

It  should  be  noted  that  these  definitions  do  not  address  the  magnitude  of 
innovation,  nor  do  they  explicitly  or  implicitly  signify  an  improvement  over  existing 
capabilities.  New  does  not  necessarily  mean  significant,  radical,  or  better.  We  might 
want  the  new  concept  to  be  “better,”  but  better  may  be  relative,  depending  on  the 
scenario.  How  much  of  an  improvement  a  new  concept,  technology,  or  doctrine  provides 
should  be  demonstrated,  in  part,  by  an  experiment. 

The  impact  of  a  given  innovation  ought  to  be  considered  within  an  appropriate 
context.  The  fact  that  magnitude  is  relative  should  not  be  forgotten,  and  degrees  of 
magnitude  expected  by  innovation  or  to  be  demonstrated  in  experiments  need  to  be 
clearly  stated.  Experimentation  helps  to  discern  the  better  from  that  which  is  simply  new. 
A  principal  objective  of  the  evaluation  process  is  to  judge  the  magnitude  or  degree  of 
improvement  or  change  that  an  advanced  warfighting  concept  may  offer  -  and  in  which 
circumstances  the  change  might  produce  significant  improvement.  One  should  not  lose 
sight  of  the  fact  that  simply  being  new  is  all  that  a  theater  commander  may  need.  Thus, 
the  advantages  of  the  newness  should  be  one  of  the  results  of  an  experiment. 


II-3 


Military  innovations  can  be  assessed  at  three  different  levels:  tactical, 
operational,  and  strategic.  Each  may  produce  a  significantly  different  perception. 
Stephen  P.  Rosen,  in  Winning  the  Next  War:  Innovation  and  the  Modem  Military,  offers 
the  following  summation  concerning  the  nature  and  magnitude  of  innovation: 

A  major  innovation  involves  a  change  in  the  concepts  of  operation  of  that 
combat  arm,  that  is,  the  ideas  governing  the  way  it  uses  its  forces  to  win  a 
campaign,  as  opposed  to  a  tactical  innovation,  which  is  a  change  in  the 
way  individual  weapons  are  applied  to  the  target  and  environment  in 
battle.  A  major  innovation  also  involves  a  change  in  the  relation  of  that 
combat  arm  to  other  combat  arms  and  a  down-grading  or  abandoning  of 
older  concepts  of  operation  and  possibly  a  formerly  dominant  weapon. 

B.  EXPERIMENTATION 

As  is  the  case  with  innovation,  the  terms  experiment  and  experimentation  are  used 
equally  loosely  to  explain  varied  approaches  intended  to  achieve  new  force  capability. 
Experiments  are  employed  extensively  in  both  the  physical  and  the  social  sciences.  At 
one  extreme,  physical  scientists  like  physicists  and  chemists  employ  experiments  in 
sterile  laboratories  using  tight  controls  and  measuring  phenomena  with  great  precision. 
The  social  scientist  conducts  experiments  in  a  more  natural  setting.  Still,  the  experiment 
designs  of  the  physical  and  social  scientist  have  a  great  deal  in  common.  Military 
experiments  are  more  aligned  with  the  untidy  world  of  the  social  scientist;  knowing  this 
will  help  us  to  understand  how  to  prepare  independent  assessments  of  experimental  and 
quasi-experimental  designs  that  will  assess  the  effectiveness  and  suitability  of  advanced 
warfighting  concepts  before  they  are  transformed  into  joint  or  Service  doctrine. 

Although  the  process  of  assessing  the  outcome  of  experiments  that  explore 
advanced  warfighting  concepts  bears  a  similarity  to  operational  test  and  evaluation 
(OT&E),  there  are  distinct  and  significant  differences.  OT&E  is  the  field  test,  under 
realistic  combat  conditions,  of  any  item  of  (or  key  component  of)  weapons,  equipment,  or 
munitions  for  the  purpose  of  determining  the  effectiveness  and  suitability  of  the  weapons, 
equipment,  or  munitions  for  use  in  combat  by  typical  military  users;  and  the  evaluation  of 
the  results  of  such  test.  Such  formal  activity  carries  with  it  the  requirement  for  test  and 
evaluation  plans,  with  reports  (often  to  Congress).  Assessments  of  proposed  or  potential 
warfighting  concepts  are  much  more  fluid  with  the  goal  of  observing  the  production  of 
imaginative  thinking.  Should  it  come  to  pass  that  new  acquisitions  are  planned  as  a  result 
of  the  experimentation,  then  formal  OT&E  becomes  a  consideration. 
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Experiments  do  several  things:  1)  they  can  support  heuristically  guided 
investigations  aimed  at  concept  exploration;  2)  they  can  support  a  goal-seeking  trial  and 
error  procedure  leading  to  implementation  or  performance  improvements;  3)  they  can  be 
used  to  test  requirements  or  to  characterize  performance  of  a  proposed  advanced 
warfighting  concept;  and  4)  when  applied  to  characterize  the  performance  of  competing 
alternative  warfighting  concepts,  can  be  used  for  purposes  of  comparative  evaluation. 
The  first  type  supports  discovery ,  the  second  supports  invention,  and  the  third  or  fourth 
types  support  evaluation.  Experiments  are  most  useful  in  testing  requirements  or 
characterizing  performance  dealing  with  proposed  advanced  warfighting  concepts. 
Military  planning  and  development,  however,  can  benefit  from  any  type  of  experiment, 
depending  on  the  purpose  of  the  inquiry,  the  need,  and  the  context  of  proposed 
employment  of  the  new  concept. 

Overall,  experiments  are  about  acquiring  knowledge  of  sufficient  quality  to 
inform  decision  makers,  those  who  must  create  or  approve  innovative  doctrine  or  initiate 
acquisition  programs.  The  ultimate  purpose  is  to  improve  military  effectiveness.  The 
experiment  is  the  centerpiece  of  the  experimental  research  method.  But  experimental 
research  is  only  one  of  several  styles  of  scientific  inquiry,  which,  in  turn,  is  only  one  form 
of  knowledge  acquisition.  The  scientists’  definition  of  experiment  relies  heavily  on 
controls:  control  of  extraneous  independent  variables  and  observation  of  outcomes.  The 
nature  of  warfare  and  of  field  trials,  however,  almost  certainly  limits  or  excludes  heavy 
reliance  on  controls  and  the  control  of  extraneous  independent  variables  as  would  be 
done  by  a  scientist.  Moreover,  one  must  be  aware  that  the  presence  of  a  free-playing 
opponent  and  the  wide  range  of  environmental  and  political  conditions  attendant  to  war 
make  suspect  (but  don’t  necessarily  invalidate)  the  requirements  or  performance 
characteristics  of  any  military  experiment. 

Lacking  the  ability  to  conduct  military  warfighting  experiments  with  scientific 
rigor,  independent  assessments  of  new  warfighting  concepts  are  required. 

The  following  definitions  from  the  American  Heritage  Dictionary  of  Science  are 
useful  for  understanding  military  warfighting  experiments: 

.  Experiment,  n:  an  operation  conducted  to  discover,  test,  or  illustrate  some 
fact  or  phenomenon,  especially  one  in  which  some  hypothesis  is  tested  by  the 
manipulation  of  one  variable  in  a  controlled  system. 

.  Experimentation,  n:  the  act  or  process  of  experimenting. 

.  Note:  the  experimental  method  is  often  confused  with  the  scientific  method, 
but  actually  experimentation  is  only  part  of  the  latter. 
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Since  these  definitions  are  too  highly  structured  to  easily  define  the  scope,  extent, 
and  flexible  nature  of  inquiries  into  warfighting  methods,  we  have  modified  them  for  use 
in  understanding  the  scope  of  experiment  assessments,  as  follows: 

.  Experiment,  n:  the  act  of  attempting  an  innovative  method  of  operation, 
especially  to  assess  its  feasibility,  evaluate  its  utility  or  suppo liability,  and 
determine  its  limits.  This  may  include  attempts  using  new  systems, 

technologies,  doctrine,  or  combinations  of  any  of  the  three. 

.  Experimentation,  n:  the  process  of  conducting  experiments,  especially  when 
successive  experiments  are  conducted  to  improve  the  performance  or 
understanding  of  an  innovative  method  of  operation. 

The  above  definitions  were  chosen  to  include  the  implicit  demands  for  controlling 
independent  variables,  for  predicting  outcomes,  and  the  need  to  conduct  repetitive  trials 
to  achieve  reliability  —  all  of  which  are  necessary  to  some  degree  for  a  successful 
experiment. 

Scientific  inquiry  is  set  apart  from  other  methods  of  knowledge  acquisition  by  the 
ability  to  arbitrate  between  competing  theories  using  a  systematic  and  objective  method. 
This  ability  stems  from  testing  alternative  explanations  —  comparing  the  outcomes  of  an 
experiment  to  operational  needs  and  definitions.  Given  that  we  cannot  control  many  of 
the  variables  in  a  warfighting  experiment  as  would  be  done  in  a  scientific 
experimentation  setting  (nor  could  we  easily  afford  the  cost  of  repetitive  efforts  seeking 
to  demonstrate  the  same  outcome),  how  then  can  one  objectively  arbitrate  between 
alternative  explanations?  The  answer  lies  in  applying  a  combination  of  techniques  from 
scientific  experimentation  and  independent  and  objective  testing.  The  ideas  of 
formulating  hypotheses,  designing  a  test  with  an  eye  to  internal  and  external  validity  (or 
at  least  an  understanding  of  the  test’s  shortcomings  with  respect  to  validity),  and 
objective  measurement  and  interpretation  of  results  can  and  should  be  incorporated  into 
appropriately  designed  experiments. 

C.  JOINTNESS 

Before  achieving  statutory  legitimacy  shortly  after  World  War  II,  the  Joint  Chiefs 
of  Staff  functioned  as  a  committee  of  co-equal  Service  chiefs  with  no  one  in  charge. 
When  a  chairman  was  later  authorized,  he  had  no  vote.  A  series  of  legislative  reforms 
culminating  in  the  Goldwater-Nichols  Act  of  1986  has  given  increasing  authority  to  the 
Chairman  of  the  Joint  Chiefs  and  to  the  legally  authorized  joint  commands.  Still,  many 
joint  activities  reflect  the  earlier  meaning  of  joint:  a  committee  of  co-equal  Service 
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representatives.  Joint  sometimes  refers  to  the  statutory  joint  offices  including  the 
Chairman  and  the  Joint  Staff,  the  unified  commands,  subordinate  unified  commands,  and 
joint  task  forces.  Implicit  in  this  reference  is  the  fact  that  these  organizations’ 
responsibility  is  at  the  strategic  and  operational  levels  of  war  as  opposed  to  the  tactical. 
Another  meaning  of  joint  is  as  indicated  in  programs  of  interest  to  more  than  one  Service, 
e.g.,  the  Joint  Strike  Fighter.  Training  exercises  that  are  conducted  to  stress  the  interfaces 
between  two  Service  headquarters  are  often  incorrectly  referred  to  as  joint  exercises 
rather  than  as  Service  interoperability  exercises. 

The  Goldwater-Nichols  Act  of  1986  aimed  at  bringing  matters  into  sharper  focus. 
As  a  result  of  that  Act,  the  Chairman  of  the  Joint  Chiefs  of  Staff  received  the 
responsibility,  subject  to  the  authority,  direction,  and  control  of  the  president  and  the 
secretary  of  defense,  to  develop  doctrine  for  the  joint  employment  of  the  armed  forces,  to 
formulate  policies  for  the  joint  training  of  the  armed  forces,  and  to  coordinate  the  military 
education  and  training  of  the  armed  forces. 

Regardless  of  the  statutory  definition  of  authority  to  develop  joint  doctrine  and 
oversee  joint  training,  jointness  means  different  things  for  different  levels  of  the 
command  structure.  Unified  commands  and  joint  task  forces  deal  with  strategic  or 
theater  operational  issues,  and  are  increasingly  trying  to  employ  joint  doctrine  in  their 
pursuit.  The  Service  components  engaged  in  combat  or  in  operations  other  than  war 
primarily  are  concerned  with  the  tactical  problems.  The  latter  tend  to  employ  Service 
doctrine  in  cooperation  with  each  other. 

Within  the  current  context,  certain  formal  definitions  taken  from  official  joint 
publications  must  be  kept  in  mind.2  The  training  community  has  provided  some 
important  definitions  that  are  helpful  to  understanding  the  differences.  Most  important  is 
the  distinction  between  component  interoperability  training,  a  type  of  Service  training, 
and  joint  training.  Often,  component  interoperability  exercises,  typically  conducted  in 
the  tactical  time  frame,  are  incorrectly  called  joint  exercises.  Apparently  only  a  minor 
semantic  infraction,  it  is,  however,  at  the  heart  of  significant  miscommunications.  Of 
similar  importance  is  the  common  practice  of  conducting  joint  exercises  concurrently 
with  single-Service  or  component  interoperability  exercises,  forcing  the  joint 


2  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms,  Joint  Pub  1-02,  23  March  1994, 

and  Joint  Training  Policy  for  the  Armed  Forces  of  the  United  States,  CJCS1  3500.01,  November  21, 
1994. 
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headquarters  to  train  in  the  tactical  time  frame,  as  opposed  to  the  operational  or  strategic 
where  it  normally  functions. 

The  major  training  categories,  by  DoD  definition,  are  as  follows: 

.  Military  Training:  The  instruction  of  personnel  to  enhance  their  capacity  to 
perform  specific  military  functions  and  tasks;  the  exercise  of  one  or  more 
military  units  conducted  to  enhance  readiness  or  ability  to  conduct  military 
operations  other  than  war.  Training  has  three  components:  Service,  joint,  and 
multinational. 

.  Service  Training:  Military  training  based  on  Service  policy  and  doctrine  to 
prepare  individuals  and  interoperable  units.  Service  training  includes  basic, 
technical,  operational,  and  component  interoperability  training.  Component 
interoperability  training  can  be  the  result  of  either  combatant  commander  or 
Service  initiative. 

.  Component  Interoperability  Training:  Operational  training  in  which  more 
than  one  Service  component  participates.  Normally,  this  type  of  training  is 
based  on  CINC-based  or  Service-based  initiatives  to  improve  responsiveness 
of  assigned  forces  to  combatant  commanders.  The  purpose  is  to  ensure 
interoperability  of  combat,  combat  support,  combat  service  support,  and 
military  equipment  between  two  or  more  Service  components. 

•  Joint  Training:  Military  training  based  on  joint  doctrine  to  prepare  joint  forces 
and/or  joint  staffs  to  respond  to  operational  requirements  deemed  necessary  by 
the  CINCs  to  execute  their  assigned  missions.  Deviations  from  these  criteria 
may  be  made  at  the  discretion  of  the  respective  combatant  commander.  For 
example,  regional  exercises  focused  on  such  CINC  priorities  as  coalition 
building,  overseas  presence  and  access,  demonstrating  national  resolve,  and 
visible  support  for  allies  could  be  included  in  the  Joint  Training  Plan. 

Two  additional  definitions  are  useful  to  bear  in  mind  concerning  jointness  and  the 
preparation  and  evaluation  of  warfighting  experiments: 

.  Exercise:  A  military  maneuver  or  simulated  operation  involving  planning, 
preparation,  and  execution.  It  is  carried  out  for  the  purpose  of  training  and 
evaluation. 

.  Joint  Exercise:  Exercises  based  on  joint  doctrine  and  procedures  that  train  and 
evaluate  joint  forces  or  staffs  to  respond  to  requirements  established  by  joint 
commanders  to  accomplish  their  assigned  missions. 

These  definitional  distinctions  are  important  in  that  they  not  only  provide  a 
doctrinal  template  and  common  view,  but  also  help  to  focus  resources  and  means  to 
achieve  the  integration  of  Service  capabilities  to  reach,  as  stated  in  Joint  Vision  201 0,  full 
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jointness  institutionally,  organizationally,  intellectually,  and  technically.  By  having  a 
clear  understanding  of  what  constitutes  joint  doctrine,  joint  training,  and  joint  exercise, 
we  are  better  able  to  consider  how  to  innovate  and  apply  experimentation  appropriately 
within  this  new  joint  context. 
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III.  DESIGNING  THE  EVALUATION 


A  great  many  experiments  are  being  considered  for  exploring  new  warfighting 
styles  and  methods  and  to  anticipate  the  defense  needs  of  the  early  part  of  the  next 
century.  Several  are  underway,  and  a  number  have  already  been  conducted  by  the 
Services.  In  general,  experiments  fall  into  three  categories,  according  to  their  purposes: 
invention,  discovery,  and  evaluation.  For  example,  where  one  sits  relative  to  the 
acquisition  process  would  dictate  how  one  wished  to  apply  the  types  of  experiments.  The 
most  obvious  dichotomy  is  the  differing  needs  of  those  responsible  for  deciding  what  to 
acquire  and  those  who  must  decide  how  to  implement  a  solution.  Yet  another  category  of 
decision  maker  is  the  unified  commander  who  might,  at  some  point,  need  to  employ  not 
only  a  new  system,  but  also  an  employment  concept  as  yet  untried  in  warfare. 

Expecting  a  satisfactory  assessment  outcome  from  an  experiment  requires  that 
those  designing  both  the  experiment  and  the  assessment  spend  time  at  the  beginning  of 
the  planning  process  clearly  defining  the  issues,  or  questions,  that  need  to  be  addressed 
by  the  experiment.  These  questions  will  drive  the  experiment’s  design  and  the  data 
collection.  There  will  be  many  participants  in  the  experimental  process:  operators, 
experimenters,  analysts,  and  decision  makers;  each  will  have  differing  perspectives.  The 
design  of  the  experiments  and  of  the  assessment  concept  should  provide  the  simple, 
comprehensive,  and  transparent  mechanism  to  ensure  consistency  and  focus  among  these 
varied  players. 

A.  PURPOSE  OF  AN  INDEPENDENT  ASSESSMENT 

One  of  the  characteristics  of  innovation  and  experimentation  is  the  value  in 
pushing  an  idea  to  the  breaking  point  and  learning  its  limits;  failure  is  a  virtue  in  this 
context.  Those  who  must  decide  what  to  acquire  or  what  new  warfighting  concept  to  try 
need  to  know  under  what  conditions  a  proposed  solution  can  be  expected  to  succeed  and 
under  what  conditions  it  may  fail.  Two  or  more  complementary  solutions  might  be 
required  rather  than  a  single  solution. 

On  the  other  hand,  those  who  have  been  assigned  responsibility  for  developing  a 
specific  material  solution  do  not  see  failure  as  an  attractive  option  and  usually  conduct 
evaluations  to  avoid  or  reduce  the  risk  of  failure.  Clearly,  these  two  groups  would 
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employ  experiments  quite  differently.  One  group  might  use  experiments  to  facilitate 
selection  between  alternative  implementations,  and  thus  appear  as  a  potential  threat  to  the 
advocate  of  any  specific  alternative.  Those  who  are  to  evaluate  the  outcomes  of 
warfighting  experiments  should  bear  in  mind  the  desirability  of  understanding  the 
conditions  of  failure  as  well  as  the  potential  conflicts  between  the  interested  parties. 

In  the  acquisition  world,  initial  operational  test  and  evaluation  determines  the 
operational  effectiveness  and  operational  suitability  of  a  production-representative  system 
under  realistic  conditions  (simulating  combat  conditions  as  nearly  as  possible),  and 
establishes  whether  or  not  the  users’  minimum  acceptable  operational  performance 
requirements  as  specified  in  the  operational  requirements  documents  (ORD)  have  been 
met.  Those  seeking  advanced  warfighting  capabilities  face  a  similar  need:  how  to 
understand  whether  the  warfighting  concept,  system,  or  system-of-systems  with  which 
they  plan  to  experiment  will  provide  them  with  answers  to  questions  they  have.  Those 
questions  about  how  to  create  new  or  different  warfighting  capabilities  become  the 
equivalent  of  the  ORD,  and  can  be  narrow  or  wide-ranging,  depending  on  the  problem 
being  addressed.  Despite  the  topic  of  the  experiment,  the  need  for  answers  is  no  less 
pressing  than  in  the  case  of  system  acquisition.  Similarly,  the  answers,  if  they  are  to  be 
employed  in  developing  future  warfighting  changes,  must  be  evaluated  by  skilled  and 
independent  reviewers.  An  Experimentation  Assessment  Plan  can  greatly  assist  in 
structuring  a  sound  experiment  and  its  evaluation. 

B.  DEFINING  THE  ASSESSMENT’S  SCOPE  AND  PURPOSE 

Systematic  planning  for  advanced  warfighting  concept  assessment  can  identify 
problems  and  organizational  requirements  and  help  the  force  planners  and  experimenters 
as  well  as  the  evaluators  reach  agreement  on  many  points.  Assessment  plans  are  not 
meant  to  be  directive  in  all  respects.  In  their  early  form,  they  are  conceptual  structures 
meant  to  motivate  and  focus  discussion  between  the  proponents  and  evaluators  on  such 
topics  as  evaluation  goals,  experiment  organization  and  design,  and  critical  resources 
needed  for  the  experiment  to  be  conducted.  Preparing  assessment  plans  also  facilitates 
transition  from  conceptual  thinking  to  defining  the  tasks  needed  to  begin  the  actual 
experiment’s  organization.  For  this  reason,  evaluators  need  to  be  involved  in  the  earliest 
stages  of  planning  a  joint  warfighting  experiment.  Figure  III-l  illustrates  the  steps 
involved  in  designing  an  advanced  warfighting  experiment  assessment  plan,  starting  with 
the  basic  goals,  then  identifying  key  issues  at  the  force,  unit,  and  system  levels,  and 
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finally  identifying  the  main  experimentation  concepts  and  critical  resources  needed  to 
implement  the  proposed  experiment. 
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KEY  IS  TO  ESTABLISH  LINKAGE  AMONG  EXPERIMENT  DESIGN, 
RESOURCES,  AND  ASSESSMENT  OBJECTIVES 

•  Ensure  adequacy  of  scope  of  the  experiment 

•  Identify  opportunity  costs  of  tradeoffs 

_ •  Estimate  possible  capability  improvements _ 


Figure  111-1.  Developing  an  Experimentation  Assessment  Concept 

An  assessment  plan  should  clearly  identify  those  issues  central  to  determining 
whether  the  new  warfighting  concept  has  potential  operational  advantages  and  will  have 
operational  utility  or  suitability  when  employed  by  a  force  in  the  field.  These  are  the 
Critical  Warfighting  Innovations  (CWls)  of  the  new  concept,  and  should  be  framed  as 
questions  phrased  to  pin  down  just  what  the  new  warfighting  concept  is  supposed  to 
provide.1  There  can  be  as  few  as  one  or  two  such  questions,  but  they  are  vital  because 
they  establish  the  tone  and  purpose  of  the  experiment  (what  the  new  capability  should 
demonstrate),  and  make  certain  that  everyone’s  expectations  are  similar.  They  are  the 
foundation  for  the  assessment  of  the  experiment’s  outcome. 

Defining  the  Critical  Warfighting  Innovations  requires  in-depth  examination  of 
the  dependencies  and  interactions  among  the  stated  experimentation  requirements;  the 
experiment  concept,  guidelines  and  procedures;  anticipated  operating  environments;  and 
the  technical  characteristics  of  the  subject  of  the  experiment.  An  assessment  plan’s 


1  Some  innovations  will  be  designed  for  operations  other  than  war,  and  are  not  rightly  “warfighting.” 
Despite  this,  the  purpose  of  a  military  force  is  to  fight  in  the  nation’s  defense.  For  this  reason  we  have 
chosen  to  employ  the  term  "Critical  Warfighting  Innovations  "  throughout  the  paper  for  consistency. 
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developers  must  consider  the  best  form  of  experimentation  in  order  to  collect  the  data 
needed  to  address  the  questions  underlying  the  CWls.  The  experiment’s  form  may 
involve  field  testing,  but  it  may,  in  addition  or  alternatively,  involve  modeling  or  human- 
in-the-loop,  software-in-the-loop,  and  hardware-in-the-loop  simulations.  Assessment 
plans  should  consider  the  feasibility  of  proposed  experiments  in  order  to  resolve  in 
advance  any  potential  problems  that  may  result  from  anticipated  limitations.  The  early 
identification  of  key  resources  needed  to  conduct  the  experiment  in  sufficient  time  to 
meet  lead  times  should  help  minimize  these  limitations. 

Because  innovation  is  a  continuing  and  cyclic  process,  an  assessment  plan  will  be 
a  living  document  that  will  change  over  time  as  more  information  about  the  experiment 
becomes  available.  The  plan’s  early  purpose  is  to  evoke  and  to  encourage  debate  about 
the  final  form  of  the  assessment,  to  avoid  as  many  last  minute  complications  as  possible, 
and  to  ensure  that  the  assessment  design  is  as  complete  as  possible.  When  fully 
developed,  the  plan  will  guide  the  final  evaluation  of  the  experiment’s  outcome. 
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IV.  ASSESSMENT  METHODOLOGY 


Many  factors  influence  experiment  designs,  all  under  the  umbrella  of  assessing 
the  potential  utility  and  suitability  of  a  warfighting  idea.  An  experiment’s  focus, 
however,  must  be  on  the  ability  of  the  candidate  concept  to  provide  new  warfighting 
potential  and  to  meet  the  evolving  needs  of  possible  users.  The  subject  of 
experimentation  and  its  relative  success  or  failure,  therefore,  must  be  judged  in  terms  of 
its  ability  to  meet  the  goals  set  for  it.  To  this  end,  it  is  important  to  select  the  most 
appropriate  tools  or  sites  for  conducting  the  experiment,  as  well  as  the  most  appropriate 
size  and  organization  of  the  group  or  team  that  is  trying  the  concept.  The  assessment  of  a 
candidate  entity  needs  to  contain  elements  of  uncertainty  as  would  be  faced  in  real  life. 
The  models  and  simulations  that  may  be  used  to  assist  in  the  experiment  must  be 
carefully  selected  and  must  include  considerations  of  verification  and  validation.  Finally, 
the  new  warfighting  concept  must  be  judged  in  terms  of  its  applicability  within  the 
military  structure  where  it  might  be  used,  and  changes  to  that  structure  that  its  adoption 
may  require. 

A.  INTENDED  OUTCOME  FROM  THE  ASSESSMENT  PLAN 

The  warfighting  experiment  assessment  plan  serves  to  highlight  certain  variables 
for  the  decision  makers:  the  issues  of  critical  importance  to  a  meaningful  independent 
assessment;  the  alternative  approaches  for  obtaining  the  needed  data;  and  the  complexity 
and  associated  costs  (in  broad  terms)  of  the  alternative  approaches.  The  purpose  behind 
the  assessment  plan  is  to  assist  decision  makers  in  reaching  early  agreement  upon  the 
principal  issues  to  be  addressed  and  the  assessment  approach  to  be  employed. 

B.  FRAMEWORK 

In  order  to  develop  a  cohesive  assessment  of  any  new  warfighting  concept,  it  may 
be  necessary  to  make  a  number  of  assumptions  as  to  the  eventual  working  of  the  concept 
in  its  military  application  (either  combat  or  operations  other  than  war).  These 
assumptions  may  center  on  understanding  the  operational  advantages  and  disadvantages 
of  the  application  or  on  assessing  the  quality  of  data  that  will  be  available,  not  only  to  the 
evaluators,  but  also  to  those  who  will  conduct  the  experiment,  and  the  evaluators’  ability 
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to  take  actions  based  on  those  data.  To  do  this,  the  proven  approaches  used  for 
operational  evaluation  can  help  to  frame  the  questions  to  be  answered  and  then  to  guide 
the  analysis  of  the  experiment’s  outcome,  or  apparent  outcome.  For  operational  test  and 
evaluation,  these  approaches  are  divided  into  areas  dealing  with  operational  effectiveness 
(in  essence,  answering  the  question:  will  the  concept  or  system,  under  realistic 
operational  conditions  and  used  by  typical  military  personnel,  contribute  to  a  unit  s 
overall  mission  effectiveness?),  and  operational  suitability  (can  the  concept  actually  be 
employed  and  supported  by  the  military  system,  and,  if  not,  how  must  the  system  or 
concept  be  altered  to  accommodate  the  concept’s  implementation?). 

To  assess  proposed  advanced  warfighting,  however,  a  broader  and  less 
constrained  approach  than  the  operational  effectiveness  and  operational  suitability  criteria 
used  in  planning  operational  test  and  evaluation  for  a  major  system  acquisition  is  needed. 
The  more  free-ranging  goals  of  joint  warfighting  experimentation  or  experiments  requires 
that  we  adopt  a  flexible,  but  parallel,  approach  to  understanding  the  value  of  new 
concepts.  To  examine  advanced  warfighting  concepts,  we  believe  that  evaluators  should 
frame  their  concerns  in  two  assessment  areas:  operational  potential  and  operational 
supportability.  We  shall  use  these  latter  terms  to  describe  a  generic  evaluation 
framework. 1 

The  operational  potential  issue  area  predominantly  involves  the  warfighting 
unit’s  ability  to  use  the  advanced  concept  to  organize,  coordinate  resources  and 
capabilities,  plan  missions,  integrate  components’  plans,  achieve  coordination  with  the 
various  approval  authorities,  issue  plans,  carry  out  the  plans,  survive,  and  recover. 

The  operational  supportability  issue  area  relates  to  the  ability  of  a  concept  when 
employed  by  a  joint  force  commander  to  support  that  commander’s  needs  in  terms  of  the 
human  requirements  for  its  use  and  the  user’s  ability  to  maintain  an  acceptable  level  of 
readiness,  to  interoperate  with  cooperating  systems  and  commands,  and  to  operate  in  a 
congested  environment  or  in  an  environment  where  an  enemy  is  trying  actively  to  disrupt 
operations.  In  addition,  the  experiment  should  yield  certain  data  necessary  for 
determining  how  manpower  intensive  and  logistically  demanding  the  concept  will  be 


1  Another  reason  for  using  different  assessment  terminology  is  to  avoid  creating  the  impression  that  the 
concept,  system,  or  system-of-systems  in  the  experiment  is  involved  in  operational  test  and  evaluation. 
OT&E  is  a  much  more  formal  process  intended  to  satisfy  a  number  of  legal  requirements  and  to 
support  decisions  about  acquisition  milestones.  Should  the  object  of  the  experiment  eventually  be  part 
of  OT&E,  its  operational  effectiveness  and  operational  suitability  would  be  assessed  in  light  of  the 
then-current  requirements. 
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when  brought  to  operational  use.  Moreover,  substantial  warfighting  changes  will  demand 
accompanying  structural  alterations  having  a  bearing  on  force  structure,  support  agencies, 
budgets,  training,  and  the  like.  The  nature  and  impact  of  these  alterations  must  be 
understood  as  well. 

An  assessment  of  advanced  warfighting  concepts  should  not  be  organized  in  such 
a  way  as  to  challenge  the  authority  or  correctness  of  those  charged  with  seeking  or 
developing  the  new  capability.  Rather,  it  is  aimed  at  ensuring  that  the  experiment’s 
requirements  are  communicated  in  a  manner  that  those  conducting  the  experiment 
understand,  and  that  allows  a  meaningful  assessment  of  the  outcome  of  the  experiment. 

1.  Assessment’s  Overall  Design 

The  starting  point  for  this  methodology  is  the  statement  of  goals  or  capabilities  or 
the  operational  changes  to  be  demonstrated  by  the  concept  or  entity  that  is  the  subject  of 
the  experiment.  The  intent  is  that  the  experiment  should,  in  so  far  as  possible,  directly 
address  those  CINC  or  JTF  needs,  assess  the  degree  of  their  satisfaction,  or  indicate  other 
areas  to  be  explored  to  determine  the  concept’s  feasibility.  Although  a  number  of 
constraints  may  limit  achievement  of  this  intent,  proper  design  of  the  assessment  issues 
will  lead  to  understanding  an  experiment’s  objectives,  its  success  at  meeting  them,  and 
the  ability  of  the  subject  of  the  experiment  to  provide  a  certain  capability  not  previously 
available. 

The  experiment’s  design  (and  its  assessment)  should  be  built  upon  the  Critical 
Warfighting  Innovations  described  in  Chapter  III,  and  should  be  aimed  at  a  logical 
progression  of  topics  and  decisions: 

.  Focusing  on  the  warfighting  concept’s  accomplishments  as  opposed  to 
performance  parameters. 

.  Describing  unresolved  mission  deficiencies  yet  to  be  evaluated  or  to  be  the 
subject  of  follow-on  experiments. 

.  Describing  how  the  concept  that  is  the  subject  of  the  experiment  addresses 
potential  mission  deficiencies. 

.  Answering  the  question,  “Did  the  experiment  succeed  in  providing  sufficient 
data  from  which  to  reach  conclusions  about  the  subject’s  performance?” 

.  Understanding  what  is  adequate  for  experiment  success. 

.  Understanding  whether  the  subject  of  the  experiment  performed  adequately 
(that  is,  one  must  understand  how  to  measure  success  in  a  given  case). 
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It  is  important  to  bear  in  mind  the  difference  here  between  the  experiment  s 
failure  to  provide  the  information  it  was  supposed  to  collect,  and  the  warfighting 
concept’s  failure  to  meet  expectations  as  the  subject  of  the  experiment. 


An  important  major  area  to  be  addressed  along  with  operational  potential  and 


operational  supportability  is  an  experiment’s  adequacy,  and  whether  or  not  the 
experiment’s  venue  or  environment  will  provide  the  information  needed  not  only  for  the 
independent  assessment,  but  also  for  the  final  decision  maker.  The  Assessment  Plan  is 
the  appropriate  mechanism  to  establish  the  hierarchical  structure  for  the  evaluation,  as 
depicted  in  Figures  III-l  and  IV-1.2  In  Figure  IV-1,  the  most  all-encompassing  issues  to 
be  weighed  are  associated  with  the  topmost  block  (1.0),  to  determine  whether  the 
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Nrw  Capability 


The  numbers  in  the  boxes  correspond  to  the  evaluation  issues  discussed  in  Chapter  IV  Blocks  2.0  Operational  Potential,  and 
3  0,  Operational  Supportability.  are  the  assessment  areas  The  lower  level  blocks  define  the  specific  assessment  issues 


Figure  IV-1.  Notional  Advanced  Warfighting  Concept  Assessment  Issues 


2  Of  course,  a  given  experiment’s  assessment  structure  would  depend  upon  the  nature,  extent  and 
purpose  of  its  subject’s  intended  use.  The  warfighting  innovation  example  on  which  the  assessment 
description  is  built  is  a  hypothetical  operational  concept  and  its  supporting  systems.  The  operational 
warfighting  concept  is  a  software  intensive,  distributed,  collaborative  planning  system  supported  by 
worldwide  communications.  The  topics  to  be  explored  in  an  actual  experiment  might  not  resemble 
those  in  this  example. 
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command  can  derive  an  advantage  from  employing  the  new  warfighting  concept.  If  the 
assessment  cannot  be  made  at  this  first  level  (and  likely  it  cannot),  then  subsequent 
analyses  must  explore  issues  farther  down  (Blocks  2.0  and  3.0)  to  determine  the  causes 
and  the  significance  of  the  advantages,  shortfalls,  or  problems  noted.  It  is  especially 
important  when  a  warfighting  concept  or  system  is  pushed  to  failure  while  in  an 
experiment  or  test  that  there  be  some  type  of  assessment  structure  so  that  one  may  follow 
the  issues  of  greatest  importance  until  one  is  able  to  define  the  point  at  which  failure 
occurred  and  the  reasons  why. 

2.  Specific  Assessment  Areas 

Using  the  statements  of  need  or  the  requirements  expressed  by  the  potential  users, 
such  as  a  unified  command  or  a  JTF,  the  next  task  is  to  identify  a  set  of  assessment  issues 
(AI).  The  AIs  will  highlight  a  small  number  of  issues,  typically  fewer  than  ten,  whose 
satisfactory  resolution  will  provide  the  information  needed  to  determine  the  advanced 
warfighting  concept’s  operational  potential  and  operational  supportability .  From  the 
answers  to  questions  posed  by  the  AIs ,  one  may  then  develop  an  assessment  of  the  value 
of  the  potential  new  capability.  Good  judgment  is  necessary  in  preparing  AIs ,  since 
assessing  the  importance  of  many  of  these  issues  is  likely  to  be,  at  least  partially, 
subjective.  From  experience,  we  know  that  there  is  no  unique  set,  and,  with  the  task  of 
studying  a  proposed  new  warfighting  capability,  the  need  for  sound  preparation  is  even 
more  crucial  than  for  an  operational  test  and  evaluation  where  the  requirements  should  be 
clearly  stated  in  approved  requirements  document  at  the  outset. 

Moreover,  to  structure  requirements  or  issues  in  any  sort  of  meaningful  hierarchy, 
one  must  estimate  the  relative  impact  of  each  upon  mission  accomplishment.  Toward 
this  end,  the  independent  assessment  team  must  review  all  possible  analyses  or  studies  of 
the  problem  conducted  while  the  need  for  the  new  capability  has  been  debated  or 
discussed.  It  is  unlikely  (and  unnecessary)  for  all  members  of  the  assessment  team  to 
agree  initially  on  a  single  set  of  issues  at  this  stage.  The  earliest  distribution  of  a  coherent 
set  of  AIs  serves  as  the  basis  for  establishing  broad  agreement  with  the  potential  users  on 
the  central  focus  of  the  assessment. 

Several  factors  underlie  the  inclusion  of  any  particular  assessment  issue.  The 
issue  must  be  demonstrably  important  to  the  experiment  subject’s  operational  capability. 
Second,  it  must  highlight  for  assessment  a  design  feature,  technology  application, 
operational  procedure,  or  system  interface  unproven  in  the  expected  application  or  in  the 
relevant  mission  scenario.  Finally,  the  subject  of  the  issue  should  be  observable  during 


IV-5 


the  experiment,  for  it  does  little  good  to  state  issues  that  cannot  be  seen  by  those  who  are 
the  assess  the  outcome.  The  intent  of  an  assessment  plan  for  an  advanced  warfighting 
capability  is  to  produce  a  structure  of  Ah  and  supporting  issues  that  are  clearly  based 
upon  mission  and  user  requirements,  exhibit  inherent  reasonableness,  and  encompass  the 
known  risks  to  operational  potential  and  operational  supportability. 

C.  ILLUSTRATIVE  OPERATIONAL  POTENTIAL  AND  OPERATIONAL 
SUPPORTABILITY  ISSUES 

This  discussion  explores,  in  rather  general  terms,  the  issues  that  affect  Block  2.0, 
Operational  Potential,  and  Block  3.0,  Operational  Supportability,  as  illustrated  in  Figure 
IV- 1,  as  well  as  the  supporting  issues  farther  down  on  the  diagram.  The  questions  posed 
here  for  each  of  the  areas  are  representative,  and  not  meant  to  exclude  any  other  questions 
or  legitimate  lines  of  inquiry  that  may  arise  during  experiment  preparation,  conduct,  or, 
for  that  matter,  the  assessment  preparation  itself. 

1.  Overall  Issues 

These  issues  relate  to  blocks  2.0  and  3.0,  and  are  the  highest  level  questions  that 
concern  the  advanced  warfighting  concept’s  capabilities. 

For  operational  potential,  will  the  concept  support  the  joint  force  commander’s 
mission  requirements  and  operational  goals?  Can  the  warfighting  concept  do  something 
new.  or  can  it  make  current  practices  perform  better,  faster,  or  at  lower  cost?  Does  the 
concept  under  consideration  resolve  the  deficiency  originally  identified?  Normally, 
much  of  the  information  to  answer  these  questions  will  be  provided  by  analyses  of  the 
issues  relating  to  Blocks  2.1,  2.2, 2.3,  and  2.4. 

For  operational  supportability,  is  the  concept  suitable  for  use  by  a  force  assigned 
a  given  task  or  mission,  or  can  the  joint  force  commander  and  any  reach-back  capability 
he  may  be  able  to  use  have  the  resource  management  capability  to  support  the  deployed 
force  in  its  mission  tasking?  Can  current  systems  and  practices  support  the  new  concept? 
What  changes  may  be  needed?  These  questions  usually  will  be  answered  by  analyses  of 
the  issues  relating  to  Blocks  3.1,  3.2,  3.3,  3.4,  and  3.5. 

Again,  the  specific  issues  shown  here  may  not  apply  to  any  given  experiment,  and 
evaluators  need  to  be  prepared  to  define  appropriate  areas  for  each  assessment. 
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a.  Operational  Potential  Assessment  Area  Issues 

1 .  Mission  Planning  (Block  2.1)  includes  those  factors  that  allow  the  successful 
incorporation  of  the  combatant  commands  as  well  as  the  JTF  headquarters  in 
compiling  operational  plans  as  well  as  the  support  of  COA  development  and 
assessment;  this  topic  includes  the  ability  to  gain  access  to  data  files  needed 
in  plan  preparation.  In  addition,  this  area  deals  with  the  ability  of  planning 
systems  to  facilitate  approval  authorities’  plan  review  actions  and  to  improve 
the  ability  of  Service  and  Agency  liaison  offices  to  expedite  deconfliction  of 
certain  requirements  and  collection  of  other  information  needed  by  planners. 

2.  Mission  Execution  (Block  2.2)  issues  are  those  intended  to  measure  changes 
in  timeliness,  in  increased  rehearsal  opportunities,  and  in  the  ability  of  the 
commander  and  his  staff  to  monitor  and  control  mission  execution,  or  in  the 
ability  to  bring  forces  to  bear  at  given  locations. 

3.  Dispersed  Operations  (Block  2.3)  issues  cover  the  ways  by  which  the  new 
concept  being  evaluated  helps  or  hinders  participants  at  disparate  locations, 
overt  and  covert,  to  participate  in  operations. 

4.  C4I  (Block  2.4)  issues  deal  with  gaining  an  understanding  of  the  ability  of 
the  new  concept  or  system  to  take  advantage  of  command,  control, 
communication,  computer  and  intelligence  resources  and  technology  to 
enhance  operations.  This  includes  determining  how  well  the  system  can  use 
established  networks  and  how  adaptable  such  networks  are  to  the  functioning 
of  the  new  concept’s  employment. 

b.  Operational  Supportability  Assessment  Area  Issues 

1.  Equipment  Deployability  (Block  3.1)  will  evaluate  the  degree  to  which  the 
concept  requires  and  how  well  or  how  poorly  the  software  and  hardware 
components  of  new  systems  or  organizations  tolerate  being  moved  by  air, 
land,  or  sea  to  dispersed  locations,  then  to  be  set  up,  and  used.  This  includes 
movement  to  isolated  locations  and  to  areas  that  present  a  wide  variety  of 
environmental  factors  such  as  cold,  heat  and  humidity,  dust,  and  variable 
electric  power. 

2.  MANPRINT  (Block  3.2)  addresses  the  assessment  in  terms  of  the  human 
requirements  needed  for  employment.  In  addition,  this  area  covers  the 
changes  in  manning  within  units  that  will  be  employed,  the  extent  and 
difficulty  of  training  necessary,  and  the  adequacy  of  operating  procedures 
that  the  users  require. 

3.  Interoperability  (Block  3.3)  inquires  into  the  consistency  of  the  innovation  to 
link  any  disparate  planning  or  operating  locations,  the  ability  to  simulate 
missions  for  analysis  and  to  deliver  complete  plans  or  sections  of  plans  to 
the  locations  that  require  them.  Under  this  evaluation  issue,  the  subject  of 
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access  control  might  evaluate  the  ability  of  the  system  to  support  a 
commander’s  or  planning  session  leader’s  need  to  closely  control  who  can 
participate  and  the  susceptibility  of  the  system  to  inadvertent  changes  in  the 
work  being  done  by  participants.  This  area  of  assessment  seeks  to 
understand  how  well  or  how  poorly  the  commander  and  his  staff  can  deliver 
a  disciplined  product  that  can  be  the  basis  of  an  effective  military  operation. 

4.  Availability  (Block  3.4)  will  evaluate  the  reliability  and  maintainability  of  the 

systems  that  compose  the  experiment  to  understand  whether  reliability  and 
maintainability  factors  fall  within  the  requirements  desired  by  the  users,  and 
the  effects  of  reliability  on  potential  operations  or  C2.  This  area  will  also 
seek  to  understand  how  well  the  system  can  be  supported  by  the  military 
logistic  structure  or  the  supporting  measures  that  will  be  needed. 

5.  System  Security  and  Susceptibility  (Block  3.5)  is  an  important  area  that  will 
establish  for  the  users  the  trust  that  they  need  to  have  in  systems  before  they 
can  use  them  effectively.  This  area  covers  both  the  ability  to  function  in  the 
face  of  electromagnetic  interference  (EMI),  both  natural  and  man-made 
(including  the  ability  to  work  in  crowded  electromagnetic  environments  of 
war  or  crisis  times).  The  second  topic  will  seek  to  determine  the  ability  of 
the  planning  and  supporting  networks  to  resist  attempts  at  penetration  from 
any  source  as  well  as  attempts  at  enemy  countermeasures.  This  issue  also 
will  evaluate  the  ability  of  the  system  to  provide  secure,  protected,  non- 
locatable  connectivity  for  dispersed  planning  sites. 

2.  Subordinate  Supporting  Issues 

Subordinate  issues  as  illustrated  in  Figures  III- 1  and  IV- 1  are  the  lowest  level  and 
most  specific  questions  to  be  answered.  Supporting  issues  address  those  factors  that 
appear  to  be  most  critical  to  understanding  a  concept’s  capabilities.  Those  in  Figure  IV- 1 
and  discussed  below  are  meant  to  be  illustrative  examples. 

The  following  are  examples  of  issues  and  supporting  topics  (See  Appendix  B  for 
a  more  fully  developed  set  of  supporting  topics  for  a  potential  generic  distributed  mission 
planning  capability). 


3.  Examples  of  Assessment  Issues 

a.  For  the  Distributed,  Collaborative  Planning  Concept  or  System: 
(Issue)  2.1  Mission  Planning  (See  Figure  IV- 1) 
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How  easily  usable  was  the  information,  and  could  it  be  integrated  into 
plans  and  operations  as  needed? 

(Supporting  Topic)  2.1.1  Does  a  system  adequately  support  the  commander  as 
well  as  any  geographically  distributed  components? 

Each  node  should  have  full  and  complete  access  to  the  system,  with  such 
access  available  during  the  preliminary  inter-component  discussions  and 
subsequently  during  plan  development  and  coordination  stages. 

For  what  portion  of  the  time  available  for  planning  was  the  system  able  to 
support  work  by  each  participating  planning  staff? 

Did  the  system  provide  access  to  databases  with  information  on 
weather,  intelligence,  friendly  forces,  map  and  terrain  features,  and  similar 
topics? 

b.  For  Firepower  Delivery-related  Concept  or  System  (as  a  secondary  example): 
(Issue)  2.1  Target  Planning 

(Supporting  Topic)  2.1.1  Will  the  available  UAV  reconnaissance  capability 
provide  continuing  imagery  and  target  location  accuracy  to  allow  completion  of  fire 
plans  and  changes  as  needed  by  the  commander  ? 

(Issue)  2.2  Mission  Execution  (See  Figure  IV- 1) 

(Supporting  Topic)  2.2.1  Does  the  employment  of  the  heavy  brigade's 
firepower  from  offshore  bring  adequate  accuracy  and  rate  of  delivery  on  targets? 

(Supporting  Topic)  2.2.2  Can  the  JTF  Commander  monitor  and  control  the 
application  of  firepower  adequately? 

Is  the  commander  able  to  operate  geographically  separate  from  the 
location  of  the  artillery,  but  retain  control  and  the  ability  to  direct  precise 
targeting  and  weapon  delivery? 

D.  MEASURES  OF  POTENTIAL  AND  MEASURES  OF  SUPPORT  ABILITY 

Having  identified  a  proposed  set  of  Als  and  associated  assessment  issues,  the  next 
step  is  to  address  the  character  and  diversity  of  information  needed  to  resolve  each  issue. 
Measures  of  Potential  and  Measures  of  Supportability  are  defined  for  each  assessment 
issue  to  provide  a  yardstick,  or  set  of  yardsticks,  by  which  the  issue  s  resolution  may  be 
judged.  Where  possible,  quantitative  measures  are  desirable.  However,  quantitative 
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measures  are  not  always  possible,  particularly  with  explorations  of  advanced  warfighting 
proposals,  and  qualitative  measures  will  be  needed  as  well.  For  example,  a  qualitative 
assessment  such  as  an  expert  opinion  might  be  used  to  judge  the  utility  of  the  content  and 
format  of  information  displayed  to  a  pilot,  gunner,  or  to  the  JTF  commander  and  his  staff. 
By  defining  the  set  of  required  Measures  (at  least  one  for  each  assessment  issue),  the 
planning  team  is  then  able  to  focus  upon  the  data  requirements  in  order  to  support  each 
identified  Measure. 

E.  SOURCES  OF  DATA 

The  methodology  developed  for  the  assessment  should  consider  primary  sources 
from  which  the  data  for  measures  of  potential  and  measures  of  supportability  might  be 
obtained,  such  as  wargames,  field  trials,  manned  simulators  or  computerized  simulations 
and  modeling,  as  well  as  the  expert  opinion  of  users.  An  early  consideration  of  the 
available  sources  is  particularly  appropriate  before  experiment  resources  are  fully 
allocated.  In  addition,  some  assets  can  only  be  made  available  with  specified  lead  times 
because  of  programmed  use  for  other  projects. 

In  developing  the  utility  and  practicality  of  alternative  experiment  approaches, 
several  factors  are  considered.  These  include: 

.  The  scope  of  activity  implied  by  each  alternative.  An  experiment  of  limited 
scope  might  address  only  the  concept’s  operational  functionality,  i.e.,  its  ability 
to  carry  out  its  design  functions  when  used  in  the  field  by  typical  operators. 
On  the  other  extreme,  a  broadly  defined  experiment  might  measure  the  ability 
of  the  concept  to  accomplish  its  goals  in  a  large-scale  adversarial  “battle,’ 
referred  to  as  a  force-on-force  exercise. 

.  The  need  for  specialized  ranges  or  facilities.  Ranges  may  provide  controlled 
regions  of  air  or  ground  space  to  support  large-scale  experiments.  Further, 
ranges  have  in  place  varying  types  and  amounts  of  instrumentation  and  data 
processing  systems  that  would  be  useful  in  obtaining  and  recording  data. 

.  The  complexity  of  specialized  instrumentation  required.  Instrumentation  needs 
are  a  function  of  the  types,  quantity,  and  precision  of  data  needed  for  the 
assessment. 

.  The  number  and  skills  of  personnel  required  to  conduct  the  experiment, 
monitor  the  collection  of  data,  and  prepare  the  assessment. 

Although  a  field  exercise  is  one  method  of  obtaining  Operational  Potential  and 
Operational  Supportability  data,  the  ability  to  address  some  assessment  issues  via  such 
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direct  field  activity  may  be  limited  by  safety,  security,  test  range  constraints, 
instrumentation  limitations,  or  cost.  In  these  cases,  simulation  or  modeling  may  be  the 
appropriate  experimentation  method. 

During  development  of  the  assessment  plan,  alternative  experimentation 
approaches  are  explored  to  determine  the  feasibility  of  resolving  AIs  in  many  ways.  The 
resources  needed  to  conduct  the  alternative  approaches  are  outlined  in  broad  terms,  that 
is,  things  such  as  major  instrumentation  needs.  In  addition,  consideration  should  be  given 
at  this  point  to  the  need  and  feasibility  of  conducting  repeated  trials  to  improve 
confidence  in  the  results.  The  intent  is  to  outline  the  framework  of  required  experiment 
resources  so  that  planning  and  budgeting  can  proceed. 

F.  EXPERIMENTATION  RESOURCES  AND  LIMITATIONS 

Any  trial  of  a  new  concept  must  include  a  realistic  appraisal  of  the  limitations  of 
that  trial,  so  that  results  may  be  judged  in  a  straightforward  manner  with  as  little  self- 
delusion  as  possible.  Experimentation  plans  must  provide  for  some  such  listing  of 
shortcomings  if  for  no  other  reason  than  to  give  users  an  idea  of  what  they  must  do  next 
in  terms  of  development  or  assessment. 

Limitations  are  those  things  that  planners  anticipate  will  significantly  or 
materially  affect  their  ability  to  address  a  major  issue  (either  a  Critical  War  fighting  Issue 
or  an  assessment  issue).  In  stating  the  limiting  factors,  the  linkage  between  potential 
limitations  and  the  relevant  CWI  or  AI  needs  to  be  clearly  established,  so  that  the  decision 
makers  conduct  the  experiment  can  assess  the  limitation’s  importance  and  effect  on  the 
outcome. 

Very  often  new  concepts  being  tried  for  the  first  time  are  not  employed  in  an 
operationally  realistic  manner.  The  fact  that  there  are  great  differences  between 
structured  demonstrations  and  an  assessment  where  forces  in  the  field  are  expected  to 
employ  a  new  system  or  concept  of  operation  to  meet  operational  goals  does  not  always 
appear  clearly  in  the  planning.  One  way  to  achieve  the  goal  of  the  experimental 
concept’s  assessment  effort  is  to  use  a  series  of  events.  The  series  can  begin  with  several 
limited  exercises  and  progress  to  a  much  broader  effort  employing  a  selection  of  units 
that  will,  to  a  great  extent,  anticipate  the  activities  of  joint  force  organizations  of  the 
future. 
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APPENDIX  A 
ACRONYMS 


UAV 


Unmanned  Aerial  Vehicle 


APPENDIX  B 
SUPPORTING  TOPICS 


Note:  The  following  topic  examples  are  notional,  with  supporting  criteria 
that  are  intended  to  provoke  thought,  not  define  the  bounds  of  further 
planning  efforts. 

A.  OPERATIONAL  POTENTIAL  SUPPORTING  TOPICS 

2.1  Mission  Planning 

2.1.1  Does  a  system  adequately  support  the  commander  as  well  as  any 
geographically  distributed  components? 

Each  node  should  have  full  and  complete  access  to  the  system,  with  such 
access  available  during  the  preliminary  inter-component  discussions  and 
subsequently  during  plan  development  and  coordination  stages. 

For  what  portion  of  the  time  available  for  planning  was  the  system  able  to 
support  work  by  each  participating  planning  staff? 

Did  the  system  provide  access  to  databases  with  information  on  weather, 
intelligence,  friendly  forces,  map  and  terrain  features,  and  similar  topics? 
How  easily  usable  was  the  information,  and  could  it  be  integrated  into 
plans  and  operations  as  needed? 

2.1.2  Can  the  planning  staffs,  especially  at  any  geographically  separated 
sites,  use  the  system  to  prepare  a  plan  or  plan  segment  that  is  of  high 
enough  quality  for  the  JTF  commander  and  his  staff  to  use? 

2.1.3  Will  the  system  support  development  of  the  CO  As  suitable  for  the 
specific  intended  mission? 

Did  the  system  provide  adequate  data  (for  example,  intelligence, 
meteorological,  order  of  battle)  to  support  CO  A  development? 

Were  planners  able  to  develop  more  than  one  COA? 

Were  the  planners  able  to  assess  each  of  the  COAs? 
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Were  sufficient  and  timely  data  available  to  support  COA  assessment? 

2.1.4  What  was  the  ability  of  the  system  to  integrate  modeling  and 
simulation  with  COA  development,  mission  planning  and  mission 
operations? 

Did  the  use  of  M&S  speed  or  retard  the  flow  of  the  experiment  and  how 
did  it  affect  the  concept  under  examination? 

Could  M&S  provide  a  realistic  threat  and  terrain  picture,  and  thereby 
enhance  the  commander’s  options  and  decisions? 

2.1.5  Does  the  system  adequately  aid  in  coordination  through  each 
approval  agency  and  authority ? 

Can  commanders  and  approval  authorities  expedite  review  in  order  to 
meet  the  planning  and  time  redistribution  needs? 

Does  use  of  the  system  expedite  review  of  plans  to  allow  Service 
component  liaison  offices  to  deconflict  air  space,  fire  support,  and  similar 
requirements?  By  how  much  has  this  been  improved  in  relation  to  similar 
actions  in  previous  mission  operations? 

At  what  point  were  plan  drafts  completed,  and  were  review  and 
coordination  needed  for  the  drafts  other  than  what  were  accomplished 
during  the  distributed  planning  process? 

2.2  Mission  Execution 

2.2.1  Will  the  concept  as  subject  of  the  experiment  support  operations  in 
a  timely  manner? 

Measured  against  the  baselines  established  during  previous  exercises  or 
operations  of  a  similar  nature,  did  the  system  deliver  approved, 
coordinated  plans  in  time  for  mission  execution? 

How  many  iterations  of  the  plan  draft  were  prepared?  What  were  the 
times  they  were  completed? 

When  was  the  final  plan  ready  for  distribution? 

How  long  did  distribution  of  the  completed  plan  take?  Is  this  longer  or 
shorter  than  the  usual  distribution  time? 
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Did  the  system  deliver  plans  without  undue  delay  once  delivery  was 
directed? 

2.2.2  Does  the  distributed  system  give  the  commander  an  effective  way  of 
monitoring  plan  preparation  and  operations  from  a  remote  location? 

2.2.3  Can  the  system  support  mission  execution  and  changes  to  the  plan 
as  the  need  develops? 

2.3  Dispersed  Operations 

2.3.1  Can  the  system  support  the  interaction  of  distributed  planners,  of 
geographically  separated  subordinate  units,  and  of  the  command  and 
control  authorities? 

2.4  C4I 

2.4.1  What  C4I  advantages  or  problems  were  evident  during  the 
experiment? 

What  capacity  communication  links  are  necessary  to  fully  implement  the 
concept? 

Taking  into  consideration  the  other  data  that  must  pass  over  these  links, 
are  the  links  adequate? 

Do  the  links  provide  secure  protection  for  planning  and  data  transfer? 
(See  section  3.5,  System  Security  and  Susceptibility.) 

B.  OPERA  TIONAL  SUPPORT  ABILITY  SUPPORTING  TOPICS 

3.1  Deployability 

3.1.1  Can  the  necessary  hardware  and  software  that  are  part  of  the 
experimental  system  or  concept  be  transported  by  land  vehicles,  rotary 
winged  and  fixed  wing  aircraft,  or  on  boats  and  ships,  and  then  be  set  up 
and  function  at  the  deployment  location? 

To  what  extent  did  the  equipment  remain  functional  after  transport,  and 
could  it  be  used  effectively  for  mission  execution  at  any  dispersed  sites? 

3.2  MANPRINT  for  the  system  or  concept  components. 

3.2.1  Is  equipment  designed  in  such  a  way  as  to  make  it  suitable  for  use 
by  deploying  forces  in  the  field? 


B-3 


What  are  the  human  factors  problems  associated  with  the  system  that 
adversely  affect  its  suitability  for  use  in  the  anticipated  environments? 

3.2.2  What  unit  manning  will  be  adequate  to  support  requirements? 

How  adequate  are  the  skills  of  personnel  assigned  to  the  sites? 

3.2.3  How  effective  was  the  training? 

Was  the  training  adequate  to  support  the  new  concept’s  use  in  the 
experiment? 

3.2.4  Are  operating  procedures  adequate? 

3.3  Interoperability 

3.3. 1  Can  the  experiment  function  smoothly  using  available  systems? 

Are  “workarounds”  needed  to  transfer  data  back  and  forth? 

Do  security  standards  and  capability  lend  themselves  to  the  employment 
concept,  and  to  what  extent  must  the  participants  take  unusual  or  special 
actions  to  ensure  protection  of  plans  and  planning  factors? 

What  new  support  systems  or  capability  will  be  needed? 

3.3.2  Can  the  experimental  concept  adequately  meet  mission  simulation, 
COA  assessment  and  refinement,  and  plan  output  delivery  requirements? 

Of  the  total  number  of  interoperability  problems  recorded,  how  many  were 
attributable  to  the  new  concept,  and  how  many  to  existing  systems? 

How  many  cannot  be  attributed  to  a  specific  source? 

What  adjustments  might  be  needed  in  employing  the  new  design? 

3.4  Availability 

Is  the  operational  availability  of  key  components  adequate  to  provide  for 
extended  periods  of  time? 

3. 4. 1  What  is  the  reliability  of  the  new  system  or  components  critical  to 
concept  employment? 

Can  it  be  measured  in  terms  of  mean  time  between  operational  mission 
failures  (MTBOMF)?  [Evaluators  might  have  to  determine  the  expected 
MTBOMF  or  other  measures  of  reliability  from  sources  to  be  provided.] 
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3.4.2  Can  key  components  be  maintained  by  unit  personnel?  Is 
contractor  maintenance  and  support  to  be  needed,  and  to  what  extent  will 
this  affect  employability? 

What  was  the  mean  time  to  repair  (MTTR)/to  reconfigure  or  reconstitute 
software? 

What  effect,  if  any,  would  there  be  if  contractors  had  to  be  used  to 
■  maintain  key  components? 

3.4.3  Is  the  concept  or  necessary  supporting  system  logistically 
supportable? 

There  may  be  no  defined  criteria  for  this  issue,  or  few,  if  any,  quantifiable 
requirements.  Evaluators  must  approach  the  issue  by  attempting  to 
understand  the  range  of  supply,  transportation  maintenance,  and  related 
problems  faced  by  a  system  that  could  be  used  in  disparate  locations. 
How  will  logistics  affect  the  concept’s  utility? 

What  are  the  special  support  functions  needed,  if  any,  and  how  do  they 
affect  system  employment? 

3.5  System  Security  and  Susceptibility 

3.5.1  How  will  electronic  interference  and  a  crowded  electronic  spectrum 
affect  planning  and  mission  execution/monitoring? 

If  no  EMI  evaluations  are  planned  during  the  experiment,  there  may  be 
gratuitous  or  inadvertent  EMI  affecting  exercises  and  demonstrations  of 
which  evaluators  need  to  be  aware. 

3.5.2  To  what  extent  might  opposing  force  penetration  be  a  problem? 

What  are  its  potentials  for  security  and  protective  countermeasures,  how 
effective  are  they,  and  how  does  their  employment  affect  operational 
effectiveness  and  suitability? 

3.5.3  Are  adequate  safeguards  in  place  to  prevent  unauthorized  entry  into 
distributed  C4  nets? 

Do  the  safeguards  provide  effective  security? 
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